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ABSTRACT 

Indicators of program quality for adult basic 
education programs, developed by consultants, are assessed as they 
might be applicable to Engl i sh-as~a-Second-Language (ESL) program 
evaluation. Sample quality indicators were developed in the areas of 
program context (i.e., organizational structure of service delivery, 
participant characteristics), program process and content (including 
planning, curriculum and materials* staff qualifications), and 
program outcomes. This paper examines Pelavin Associates' sample 
outline of proposed areas in which indicators as well as sample data 
elements for the indicators will be offered. For each program area in 
which quality indicators are recommended, the following questions are 
addressed: (1) Are there unique aspects of ESL programs that should 
be taken into account in development of indicators, and what 
adjustments and adaptations to generic indicators may be necessary?; 
(2) Are the indicators appropriate for ESL programs , unneces sary , 
incomplete?; and (3) What problems or other issues would confront ESL 
programs when using these or similar indicators? Responses to the 
questions are appended to the main text. (MSE) (Adjunct ERIC 
Clearinghouse on Literacy Education) 
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This paper was commissioned by Pelavin Associates Inc. to assist them in developing indicators 
of program quality for state adult education programs. Pelavin is preparing these indicators for 
the U.S. Department of Education, Office of Vocational and Adult Education. Recent amendments 
to the Adult Education Act require that indicators be developed at the federal level by Jufy 1992 
and by each state by 1993. This paper examines Pelavin's sample outline of proposed areas in 
which indicators will be addressed and some sample data elements fd* the indicators. This paper 
is written from an ESL programming perspective. 

Pelavin has developed Sample Quality Indicators in the following arecs: 

SECTION I: PROGRAM CONTEXT 

SECTION II. PROGRAM PROCESS AND CONTENT 

Part I. Program Planning 
Part II. Program Process & Content 
Part III. Curriculum and Materials 
Part IV. Staff Qualifications 

SECTION III. PROGRAM OUTCOMES 

The following questions posed by Pelavin Associates are addressed at the end of this document 
(see Questions for Consideration) for each program area in which quality indicators are 
recommended: 

1) Are there unique aspects of ESL programs that should be taken into account in the 
development of indicators? What adjustments and adaptations to a generic set of 
indicators may be necessary? 

2) Are these indicators appropriate for ESL programs? Are any unnecessary? Should any 
be added? Which ones and why? 

3) What problems or other issues would confront ESL programs when using these or 
similar indicators? 

INTRODUCTION 

In general, the development and use of quality indicators will be welcomed by the ESL field, 
particularly in light of the new federal guidelines which will further open up the delivery of ABE 
and ESL services to other providers. However, how indicators are formulated at the state level, 
what expectations they cany, and what support will be made available for implementation will be 
the determining factors in whether indicators will be accepted as the means of improving program 
quality or viewed as another bureaucratic burden on already overstretched program staff. If the 
use of quality indicators is to be successful we need to consider the following: 

* Due to the wide diversity among ESL programs in terms of type of program, program 
setting, learner goals and funding sources, it is impossible to have a "one size fits air 
approach to quality indicators. Research has found that even among the best programs, 
there is no "one definitive ESL or Literacy program" whose practices and procedures 
should be adopted by all. 1 
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* Most programs have been operating on a shoestring. It will be difficult to compare 
programs and hold them accountable to an external set of outcome standards when some 
programs are not used to having outcome indicators of their own. Adequate funding to 
implement quality standards must be forthcoming. A sufficient period of time to raise 
programs to quality levels will be required along with technical assistance and even 
external evaluators. 

* The primary purpose of quality indicators should be to help programs define 
themselves as educationally sound systems for the delivery of ESL and literacy 
instruction. 

Section I & II of the quality indicators proposed by Pelavin can help with this definition of 
process. Section III seeks to use quality indicators to measure outcomes. 



SECTION I. PROGRAM CONTEXT 

Indicators of quality in the area of "program context" would require that ESL programs be able 
to define the context in which services will be offered. This entails an assessment of need for the 
services, a description of organizational structure of the delivery system and a definition of the 
characteristics of the participants. Pelavin suggests defining contexts through information gathered 
in the following areas: 

1 . Need For Program Services 

A. Number and demographics of target population 
Q. Literacy Levels in community 

C. High School drop-outs in community 

D. Employment related skill needs of community 

2. Organization and Structure of Delivery System 

A. Number of projects 

B. Locations and settings of projects 

C. Type of projects (e.g. ESL.GED) 

3. Characteristics of Participants by skill level 

A. Number and demographics of participants by skill level 

B. Number and demographics of participants by program type 

C. Number and demographics of participants by program setting 

RATIONALE: 

The process of defining context should begin with the development of a program mission and 
ideally a statement of the ESL program's driving philosophy. The mission statement should 
clearly and concisely define the program's purpose for existence. Yet, the statement must be 
broad enough to enable continued program growth and evolution through flexible and 
responsive programming. Once the ESL program understands and articulates its own broad 
mission, it can make more rational decisions on program direction and use of resources. 
Furthermore, the state agency can more objectively determine an applicant agency's 
appropriateness for delivering the proposed services, and determine among other things: 

if the applicant agency understands the funding agency's mission and priorities 

(which themselves must first be clearly articulated to all) 

2 
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if the applicant agency has adequately defined its target population 

if the target population is eligible for services 

to what extent the need for services exists 

how much of the need the applicant agency can realistically address 

if proposed services match the needs of the target population, and 

the extent to which proposed services address a wide range of community needs 

and learner goals 

Moreover, requiring ESL programs to provide information in these areas will strengthen the 
program's ability to make an assessment of the overall need for program services, and it will help 
them to determine their "strategic market position in relation to other providers." 2 This is even 
more important now as new providers enter the delivery system. Information required by the 
proposed indicators will help both new and existing programs to clearly define "what business" 
they are in and what services or benefits they can offer to specific constituencies. It will require 
programs to closely examine and reexamine their immediate program and environmental context. 
By doing so, they will become better equipped to determine not only who they are serving, but 
more importantly, who they are not serving and whether or not they should be. This 
information: of course, will lead to more creative and dynamic programming because it will require 
regular assessment of needs and an evaluation of service response to those needs. 

The most important aspect of ESL programs to consider is that they have ever changing 
populations. Programs should conduct periodic environmental scans to determine such 
information as: 

* Population trends 

* Economic development trends 

* Workplace trends 

* Legislative trends 3 

%. 

There are several other questions that ESL programs might want to investigate, such as what are 
the educational motivators among the target population, how does the target population spend 
its time, when are participants available for instruction, etc. Ciearly answers to these and other 
questions will provide important information to both the ESL program planners and those 
responsible for determining whether proposed sen/ices are warranted. Realistically, however, 
because of time and budget constraints, programs should only be required to demonstrate their 
understanding of the target population, the current institutional response from other providers in 
the service delivery area, and the extent of need for services by providing information such as 
the proposed Pelavin examples. 



SECTION II: PROGRAM PROCESS AND CONTENT 

Additional areas in which indicators of program quality are being considered are program process 
and content. These areas relate to: program planning, program processes and content, staff 
qualifications, and curriculum and materials. 



PART I, Program Planning 



Pelavin suggests that program planning should consist of such elements as: 

1. Community Input in program development A. Existence of an advisory board 

B. Program holds public hearings 

C. Use of a needs assessment 

D. Other sources consulted (e.g., employers, staff, etc.) 

2. Coordination activities 

A. Existence of coordination arrangements 

B. Type of coordinated activities 

3. Written operational plan 

A. Existence of plan 

8. Specific program goals and objectives consistent with state plan 

C. Plan development process 

RATIONALE: 

Program planning is a process which includes identifying needs, designing program activities, 
implementing those activities, evaluating results, and making further program decisions. This 
planning should result in a product - a written operational plan which enables all personnel to 
understand, analyze, and critique the program goals, objectives, and the strategies used to 
achieve them. It sets the framework for an educationally sound program. Ongoing analysis, 
critique, and revision of plans enable programs to remain dynamic and responsive to new needs 
and population trends, it is therefore a reasonable requirement for programs to not only be able 
to describe their process for successful implementation, but also their means of measuring that 
success within those implementation steos. Establishing indicators of program quaiity in the area 
of program planning will lead to more effective implementation and informative evaluation systems 
for both programs and funding agencies. Requiring evidence of planning will also enable both 
the program and the state agencies to determine if a program has struck a very necessary 
balance between what needs to be maintained and what needs to be changed. Although many 
programs have been in operation for a long time, it is important that ongoing reevaluation of 
program needs and activities be conducted in order to determine what activities still meet the 
target population's needs and which ones have just become ritual. In this way, program activities 
can be more targeted to current training needs, and resources can be used more effectively. 

What must be safeguarded, however, is a program's flexibility. An operational plan must not 
be so rigid or a program manager so concerned about being evaluated against the plan, that it 
cannot be amended in order to meet a more urgent or changing need. Program planners must 
remember that they are there for the learners, and policy at all levels must support that. 

Community input ir ESL programs is also important. There is a growing trend among ESL 
programs to seek out and involve representatives from the language communities that are to be 
served, the learners themselves, and other sources of input (including major employers and 
potential employers) that have an interest in the target community. Some examples of 
collaboration and meaningful input are: 

* learner and/or teacher participation in governance, program planning & evaluation, 
curriculum, etc. 

* collaborations with other literacy providers, workplace settings(unions,employers) 
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Quality indicators should encourage these types of collaborations where appropriate and 
necessary. 

PART II: PROGRAM PROCESS AND CONTENT 

Pelavin suggests quality indicators for the following activities: 

1. Recruitment 

2. Program Intake procedures 

3. Ongoing assessment methods 

4. Support services 

5. Exit and follow up procedures 

RATIONALE: 

The indicators of program quality that Pelavin suggests here are consistent with the work of other 
researchers who have identified effective and educationally sound program processes. In the 
areas of program content, Rene Lerche concluded from her work on The National Adult Literacy 
Study (N ALP) that effective adult literacy programs result from a systematic approach to program 
design and implementation. "Successful programs have been designed as total education systems 
under which there is a balanced emphasis on (1 ) clearly stated learning objectives, (2) assessment 
of learner needs and progress, (3) instructional processes, (4) guidance and counseling, and (5) 
program management and evaluation." Lerche finds that effective programs: 

* are clear about their goals 

* have measurable objectives for each program activity 

* assist learners in determining if the program is suited to their goals 

* have clear learner outcomes and standards forjudging them 

* diagnose each learner's needs and develop individual learning plans 

* tie learning objectives to instructional methods, materials, and assessment 

* provide feedback to learners on their progress and document progress 

* evaluate their program's effectiveness 4 

Aguirre International, in their U.S. Department of Education Study of Effective and Innovative 
Practices in Adult ESL Literacy Programs, also examined eight program components similar to 
the NALP: (1) community outreach (2) needs assessment, (3) program design, (4) curriculum, (5) 
approaches and methods, (6) initial assessment and progress evaluation, (7) staff development, 
and (8) support services. 6 

These components are the nuts and bolts of an ESL program. They identify who is to be served, 
how they will be recruited, how their skills will be assessed for placement and progress, what 
support services will breakdown barriers to participation, and what will happen at program exit and 
follow up. Indicators of program quality should be established in these areas in such a way 
as to demonstrate to the funding agency that the program processes have been articulated 
based on the information that was forthcoming during the planning process and needs 
assessment. The key to successful and responsive ESL program design is to strike a balance 
between what a program can offer (given its experience, resources and staff) and what the 
learners need and want. Quality indicators should be structured to help shape this process. 
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The following are some issues to consider within various program activities: 
Recruitment 

Most ESL programs will have no trouble defining their target populations, their recruitment 
methods, and their program outreach activities. Indicators of program quality should require that 
programs be able to demonstrate that they are targeting outreach efforts to learners for whom the 
program is appropriate and can show learners how the program can help them. 

Program Intake Procedures 

Program assessment procedures will, in some cases, have to be tightened up and be more 
sharply defined to meet standards. Again, because of budget constraints, many programs have 
limited resources for testing and often students are placed in class through an informal interview. 
At intake, procedures need to be set into place which allow for a balance betveen standardized 
tests (that allow for comparisons across programs) and program based alternative assessments 
that show where the learner fits into the particular program scheme. 

Individual learning plans (as a document) for low proficiency level students are not practical 
because the learners cannot articulate their goals with enough specificity to be meaningful. 
Conducting the interviews bilingually is also not possible for large programs with students from 
many different language backgrounds. While the process of helping learners articulate 
learning goals should begin at the lower levels (within the limited language available to the 
student) individual learning plans should only be required of intermediate level and 
advanced students. 

Ongoing Assessment Methods 

Programs are required by federal law to report standardized testing information as one measure 
of learner gains and program success. Unfortunately, that is the one measure which may be the 
least valid in determining those factors. To date, there is no one instrument which can be 
recommended for all program contexts. Nor is there any that is sensitive enough to record general 
learner gains in the short periods of course time in which most ESL programs operate (40-100 
hrs of non-intensive instruction). ESL programs should be allowed to demonstrate that their 
assessment methods for determining student progress and learning gains are consistent 
with the type of program that they provide, and that their methods are valid and reliable. 
Curriculum-based pre and post tests, competency based assessment or other alternative 
assessment systems which demonstrate gains in learning related to the instructional program and 
content will provide more meaningful data than scores on standardized achievement tests. 
However, it is possible to use standardized test scores to establish and validate program levels 
and determine program effectiveness among groups of learners as opposed to individual learner 
progress. The question remains which test to select and what is an appropriate sampling size and 
testing procedures. Technical assistance will surely be required. 

Support Services 

Programs should not be required to provide support services unless significant funds are 
available, rather they should be required to demonstrate their knowledge of where the services 
exist and how their participants are made aware of them. 
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Exit and Follow Up Procedures: 

Exit interviews or counseling sessions are reasonable and appropriate, but of course require a 
substantial amount of additional time and resources. 

ESL program participants are a highly transient population, and follow-up requirements ( for each 
participant) to determine program impact would be difficult to meet. However, programs should 
conduct surveys of program drop outs to determine whether the program services should be 
improved or changed. 

PART III. CURRICULUM AND MATERIALS 

Pelavin suggests indicators of program quality related to: 

1. Type of curriculum and Instruction mu d 

2. Materials and equipment used 

3. Selection and evaluation of materials and equipment 

RATIONALE: 

Information about curriculum, materials and equipment used, and how those materials are 
selected is valuable, particularly at a time when the field is changing, new trends are emerging, 
and the use of educational technologies is being encouraged. Aguirre International, has found 
that: u a strong curriculum is a conceptual framework that (1) outlines the kinds of literacy the 
program wants to achieve, (2) suggests approaches, methods, and materials; and (3) links 
classroom teaching at the various levels with assessment and evaluation. 11 * 

Since ESL learner needs and goals are broad and vary from program to program, the state 
should not necessarily mandate a standardized curriculum or mode of instruction, but guidelines 
in effective practices are appropriate. Some states have already provided direction to programs 
on methods and approaches, eg., California stresses lifeskills instruction and language acquisition 
over "language learning", other states are now identifying approaches that work best with second 
language learners (New York), and others have developed or are developing curriculum guides 
(Washington,Texas and Florida). 7 At a minimum, states should require that programs define the 
linkage between the nseds assessment and the curriculum. Programs should establish their 
curricula first in terms which are appropriate for their learner goals and then in terms of learner 
needs within those goals. States should also demand that the approach is educationally sound 
and reflects an understanding of what we know about second language acquisition, adult 
literacy and how adults learn. 

Based on a review of the literature and on input from researchers and practitioners in the field, the Aguine study showed 
that educationally sound ESL literacy lessons shared the following characteristics. They are: 

Interactive (students talk to each other and to the teacher; students are actively involved in generating writing 
and interpreting what others have written;) 

Responsive to multi-levels (students work together as a group or in pairs; activities work for different student 
levels) 

Learner<entered(the type of language and literacy taught supports the goals of the learners and builds on their 
personal strengths and their life experiences; students have the opportunity to make choices during the lesson; 
classroom activities are linked to learners' lives outside of class.) 
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Meaning-baaad and communicative (the activities that students are engaged in reflect language and literacy 
use outside of the classroom there is a point to the lesson other than "literacy practice".) 

Integrated fn reaped to language akllla (reading, writing and oral language use are connected; conversations 
and discussions lead to reading and writing and vice versa) 

They also provide a balance between activities that focus on communications and the expression of ideas and those that 
emphasize language awareness. 9 

A strong ESL curriculum will also be organized by instructional levels with specific learner 
outcomes and related language skills. Additionally, a strong curriculum will define the methods 
and materials used for implementing the program. Most importantly, the methods and materials 
used by ESL programs should reflect an understanding of how adults learn and should provide 
for differences in a learner's ability and learning styles. 

PART IV: STAFF QUALIFICATIONS 

Pelavin proposes indicators of quality in this area as follows: 

1 . Characteristics of staff 

2. Staff responsibilities 

3. Staff development 

4. Use of volunteer staff 

5. Evaluation of staff performance 

RATIONALE: 

Implementation of a quality ESL program is dependent on the staff hired to conduct program 
activities. Yet, in the past, it was a commonly held view that "if you can speak English, you can 
teach English". It is difficult to say whether this view is the result of common employment 
conditions today or the cause of them. Currently, like the field of adult literacy in general, the 
majority of adult ESL programs in the country are staffed by part time teachers, some of whom 
are trained professionals, others are trained in other fields and others are volunteers. 9 There is 
an ongoing debate regarding "who is an adult educator?** and "who educates adults?" This 
debate cries out for the professionalization of the field of adult education and the teaching of 
English to adults. The indicators which Pelavin has suggested are appropriate and further our 
goal of professionalization. It would be helpful to see minimum staffing standards related to 
training, full/part time status, staff evaluation, and staff input into program design and evaluation. 
These minimum standards must be backed by the resources necessary to implement them. 
A discussion of staff qualifications, however, must recognize the fact that there is no agreement 
in the field over appropriate qualifications for ESL literacy teachers. Minimum staffing standards 
may really work against a number of community-based organizations. Despite this conflict, if we 
are to improve quality of instruction, guidelines need to be established to assist CBOs in 
improving their staffing situations as well. TESOL's Standards and Self-Study document should 
be consulted in the development of standards for this area. 10 

Given this ongoing debate, standards for staff development will need to take a more prominent 
role for ESL programs. Staff development should be based on the needs of the instructors, the 
needs of the learners and the needs of the program. Standards will also need to be flexible 
enough to allow for changes that are occurring in staff development such as more teacner 
centered approaches, where the teacher is helped to meet his/her own goals. This 

8 
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individualization of staff development is a veiy powerful means of met ng the needs of teachers, 
learners and programs. 

Staff responsibilities is another issue that needs to be closely considered. Teachers should not 
be expected to volunteer their time beyond the paid classroom duties. The quality indicators that 
are being proposed will impact on them by requiring additional responsibilities for counseling, data 
collection ^nd program input. They should be paid for these additional responsibilities. Most 
importantly "re should be paid for planning time. Paying for planning time in itself sends a 
message to le r actitioner that their endeavor is a professional one reauiring an assessment of 
their individual s dent needs, an adaptation of the program curriculum to meet the needs and 
an integral , o* appropriate instructional and evaluation techniques and materials to implement 
the curricut «■ '.'I'ectively. 

SECTION III: PROGRAM OUTCOMES 

Pelavin has suggested indicators for the following program outcomes: 

1 . Rstantlon 

2. Educational gaina 

3. Employment 

4. Goal achievement 

RATIONALE: 

These indicators reflect the need to determine both impact on ESL learners and provide data for 
program accountability. No one would dispute the fact that accountability is necessary and 
justified. If we are designing and implementing sound learning systems then we should see 
successful outcomes. The problem then remains to identify and use appropriate indicators and 
to separate out the purpose for which the data for the indicators will be used. Ruth Nickse 
in her report, "A Typology of Family and Intergenerational Literacy Programs: Implications for 
Evaluation", identifies five levels of evaluation purpose that she applied to various types of family 
literacy programs. The following chart is an adaptation of her framework (for the purpose of 
examining Pelavin's quality indicators).' 1 



LEVELS 


LEVEL 1 : 

NEEDS ASSESSMENT 


LEVEL 2: 

ACCOUNTABILITY 


LEVEL 3: 
PROCESS 
CLAUDICATION 


LEVEL 4: 

PARTtOPANT PROGRESS 


LEVEL 6: 

PROGRAM IMPACT 


QUESTION 


Is there • need tor the 

program? 


What will the program do 
to meet tie need and how 
can M be monitored? 


How can tha program 
improva its tarvicas? 


Art participants making 
prograsa? 


What aro Ho long term effects 
of program partapants? 


DATA ELEMENTS 
TO CONSIDER 


'dtmograpWe Into, 
'program deelgn A 
contorts 


•written operatonal pten 
*proot«M» A content 
*cvmcuiL j m and 
materials 

**nputs:»*ff,comrTwnrty, 
aoVteory boenX 
tvakjator, Junolng 
souroa 


•parttetpaiion rataa 
*rat isonratee 
'program oomptouon 
•client aaiialacfon. 


•proficiency gains 
lava! gams 
*goal ettamment 


•continuing education 
•improved Job or income 
•long term goals met 



Most adult ESL programs should be able to meet at least the first four levels of evaluation by 
providing the suggested outcome data. The fifth level, long term program impact as represented 
by Pelavin's suggested "employment" related indicators may not be possible to collect accurately 
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When it is collected and reported for state reports, the information may not be correct because 
it is often second-hand information from other students. Employment related outcomes, while 
interesting, cannot truly be attributable to general ESL program, even if valid data collection in 
this area were feasible. 

The emplcyment related outcomes that Pelavin suggests are related to what Hal Beder described 
in his new book, Adult Literacy Issues For Policy and Practice as the "human capital theory of 
program impact.The human capital theory of impact provides justification for federal expenditures 
on adult literacy instruction, the theory being that improving literacy and basic skills will lead to 
11 improved worker productivity which in turn leads to improved national productivity and increased 
national wealth." Beder lists four components to human capital impact increased employment 
and increased quality of employment, increased income, reduced need for public assistance, 
continued investment by the student in further education. Beder noted several studies which 
showed human capital gains, but notes that without experimental control groups, there is no way 
to know with certainty that those gains were a result of adult literacy instruction. Beder states that 
when reasons why adults participate are considered, it is clear that most are striving to improve 
themselves. Beder recommends that The federal adult literacy education program be held 
accountable for the broad personal and social development of its clients rather than to narrow 
human capital outcomes". 12 

Many programs have valid goals that are not directly related to employment such as 
strengthening literacy practices at home (family literacy); helping learners to communicate in 
English, access services, and gain greater independence (life skills programs), promote greater 
participation in the democratic process (community literacy and civics classes)preparing to enter 
GED, college or vocational programs. 

Much of the literature maintains that increased self-esteem, stronger decision making skills, and 
effective strategies for learning how to learn are important outcomes of language and literacy 
programs. Programs that include these domains in their framework should be encouraged to 
show: 

(1) how their efforts link linguistic goafs with non-linguistic goals, 

(2) what educational opportunities are provided to reach these goals, and 

(3) how staff plans to evaluate whether (and to what extent) these goals have been met 2 

Therefore, given the fact that we cannot directly attribute human capital gains to general ESL 
programs, given the need to also impact on the social and personal goals of learners (because 
that is their primary reason for participation), and given the limited resources that programs have, 
isn't it more important to measure outcomes over which we have control i.e, retention, educational 
gains and goal achievement? These three areas are within a program's control over data, and 
outcomes can be directly attributable to program processes. 

Making comparisons of outcomes across programs, however, will still be difficult since effective 
ESL programming is a dynamic process. Outcomes are cased on individual program focus, 
learner goals, educationally sound program curricula and practices, quality of classroom 
instruction and evaluation procedures. No two programs are the same. Quality indicators should 
take this into account and allow for program-based assessment. 
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Summary 

While the use of quality indicators is appropriate, we must be realistic in our expectations. After 
conducting research on over 200 programs nominated for their promising practices, Wrigley and 
Guth have found that it is not possible to identify "one definitive program to model". In their 
forthcoming manual. Adult ESL Literacy: Issues, Approaches.and Promising Practices , the 
researchers state: 

Given the largo variety of programs and the diversity in program focus (general, workplace,famity, community 
l iteracy) and program goals (self-sufficiency; acculturation; academic...) "establishing standards" might require 
that we emphasize processes that help ensure quality instead of focusing on outcomes or products. That is, 
programs should be held accountable for having in place structures, plans, and emluation tools, that promote 
quality education. 

For accountability's sake, programs will be asked to show how they 

* plan to implement a quality program 

* "define success" (both in program terms and related to learner outcomes) 

* will evaluate program success and judge learner progress 

plan to develop their own standards and help ensure that those will be met. 

Accountability will be based on the quality of: 

1) their planning process 

2) their effort to provide quality service 

3) their flexibility and responsiveness to learner needs 

4) their evaluation efforts, geared toward both program improvement overall and assessment of learner progress 
and performance 

5) the actual changes made based on the evaluation results 

6) the establishment of quality indicators based on program experience over a reasonable time 14 

While this may not be a popular approach to take in defining program accountability, it may be 
a more realistic one (given the state of the art and the unique characteristics of ESL programs 
which on the one hand enables them to be responsive, and on the other hand impedes 
comparison of programs against each other). 

When finalizing quality indicators, it is essential that the following general characteristics of ESL 
programs be kept in mind: 

1 ) There is wide diversity among ESL programs in terms of service delivery settings, in 
terms of program focus, and in terms of funding sources. Quality indicators must be 
broad enough to capture the essence and the reality of these varied programs and 
flexible enough to enable them to be responsive to their client population and to the 
many other different requirements that they must be accountable for (funding sources, 
parent organization rules and regulations, etc.) 

2) Qualify indicators must be written in such a way that they promote responsive 
programming not inhibit it Programs must continue to be able to be responsive to the 
wide range and varied needs and goals of their learners. 

3) ESL programs have limited resources. The spirit of the indicators should be such 
that it encourages and promotes quality services to the learners. If accountability 
requirements become so cumbersome, and monitoring so rigid, as to cause programs to 
spend an unreasonable amount of valuable time and resources to meet those 
requirements, then quality will suffer. If programs are to meet and maintain a minimum set 
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of standards, then additional time, resources and technical assistance must be made 
available. 

The field is changing and quality indicators must allow for new trends to emerge and for new 
approaches to be tried. Quality indicators must take into account the creative and experimental 
nature of programming for effective and innovative ESL and literacy programs. Quality indicators 
must enable the emergence and experimentation with such trends as: 

* Participatory education: Programs in which learners have a role in program 
implementation, curriculum, and assessment 

* Learner strategies training: An approach which encourages cognitive awareness and 
control over learning 

* Educational technologies: There is still much to be learned about the potential that new 
technologies hold for language teaching 

* Functional context instruction: An approach to training which narrows learning to specific 
content areas 

The pluralistic and experimental nature of the field of adult ESL must be safeguarded or we risk 
the loss of responsive education and the ability to grow and expand knowledge and experience 
in exploring innovative and effective ways to provide instruction. 
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QUESTIONS FOR CONSIDERATION 



SECTION 1: Program Context 



1) Are there unique aspects of ESL programs that should be taken Into account In the development of Indicators? 
Wtat adjustments and adaptations to a generic set of Indicators may be necessary? 

ESL programs are unique in having a greater need for assessing information which can help them define their ever 
changing and growing populations. This ever-constant need to evaluate context and make programming decisions based 
on the context defines the very "fluid nature of ESL programming." However, this characteristic of good ESL programs 
will make it difficult to evaluate programs against each other. The immediate implication is that programs should be 
evaluated against how they have defined their contexts, what processes they have established to be responsive and what 
outcomes they have established for themselves. 15 This is surely not a politically popular implication, but it is the reality 
of the nature of ESL programs. 

Concerning information on "the number and demographics by skill level" pg.2, programs should have formal 
descriptions of proficiency levels and should define the relationships among their placement tests, instructional levels, 
and expected outcomes.' 9 Ideally, the levels would be established or correlated to nationally recognized systems. This 
can be done by either establishing common skill level definitions at the state level or enabling programs to give their own 
definitions. There are several possible resources for defining levels: the Mainstream English Training Program's Student 
Performance Level Descriptions, 17 CASAS Scores 1 * or BEST test scores. 10 Several states already have skill level 
descriptions defined by the previously mentioned systems. However, it might be more advantageous and realistic for 
programs to be required to define their own skill levels and demonstrate that they are appropriate for the type of 
learner and ESL program services that are being offered. Perhaps the first step is to establish consistency and 
progression within a program before we try to compare programs. If programs define their own skill levels, those 
definitions should clearly indicate both the general language and specific language skills that are represented at program 
entry and/or are expected at exit. 

Skill level descriptions will obviously differ between programs that focus on teaching English for iifeskills communication 
and programs that focus on workplace literacy skills! or pre-academic skills. 

2) Are these Indicators appropriate for ESL programs? Are any unnecessary? Should any be added? Which ones 
and why? 

The information requirements proposed are appropriate and should be considered baseline. Ideally, programs will find 
out more information than is being required here. 

SUGGESTED ADDITIONS: A statement of program mission. A broad mission statement will help the program determine 
how it perceives itself, how it wants to be perceived and whom it wants to serve. In this way it will help the state 
determine if proposed services are appropriate for the proposing organization. 

DROP OR ADJUST: "Literacy levels of the community" (pg.2 of this report) - change to education levels of the 
community. 

The literacy levels of the community" is confusing. Who is the community? Those that are receiving services? The 
general population? The proposed target population? How would one gather data on the literacy levels of the community? 
Perhaps this should be changed to "education data on the community", which can easily be gathered through such 
means as census data and economic development data. Information on sub populations may also be possible, such as 
the number of people in the census tract that report that they do not speak English well. 

3) Problems or other Issues which would confront ESL programs when using these or similar Indicators? 

Clear definitions of terms within the indicators will be required. For example, does "community" refer to the community 
in general or the target population? Does "skill level" mean individual language skills (reading, writing, listening, speaking) 
or does it mean general proficiency level? Perhaps skill levels would even be defined more broadly to include knowledge 
of U .S. culture or content such as workplace skills, citizenship information, knowledge of the community, etc. that a learner 
is expected to have or expects to develop in the course. 

Concerning "Characteristics of participants by skill level" (pg.2 of this report) ( data requirements on participant), 
in order to facilitate information to programs seeking data information and to avoid duplication of effort, it will be necessary 
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to have a reporting process which will facilitate easy access to information on learners and programs that are currently 
in the system. The state's annual report to the U.S. Department of Education can serve as one valuable source of 
information for local providers as well as for the Department of Education. It should also be noted that programs may not 
be familiar with sources for data requirements and some guidance or technical assistance may be necessary. 

Last, but not least, is the fact that ESL programs are generally poorly funded and primarily staffed with part-time 
personnel. Although many programs conduct the aforementioned activities, the information may not always be in a format 
that is meant for outside consumption. 

SECTION II: Program Process and Content 
PART I: Program Planning 

1) Are there unique aspects of ESL programs that should be taken Into account In the development of Indicators? 
What adjustments and adaptations to a generic sat of Indicators may be necessary? 

Although Pelavin's indicators are reasonable for this area, it can be expected that there may be many programs that 
would not be able to produce written evidence of these activities because of time and resource constraints. Good 
programs, however, do conduct these kinds of planning activities. It Is the planning and evaluation process that Is 
most Important The task of developing an overall operational plan may seem overwhelming at first, but programs should 
be encouraged to organize and compile materials produced from the planning process into a systematic operational plan 
and demonstrate how that operational plan was developed, how it will be evaluated, and how it will be revised. 
If programs establish an overall operational plan then the next logical step is technical assistance for formative and 
summative evaluations of the plan. 

2) Are these Indicators appropriate for ESL programs? Are any unnecessary?, Should any be added? Which ones 
and why? 

Pelavin suggests a need for quality indicators for Program Planning related to community input in program development, 
coordination activities, and written operational plan. Each of these areas is appropriate in that they take into account the 
who, what, where, when and why of planning. All ESL programs should conduct planning and coordinating activities. 
However, "Holding Public HearIngs M (pg. 4 of this report), may be an indicator which is significant only if the state or 
federal agency requires it. Less formal focus group discussions with current and potential clientele usually are far more 
desirable and productive. Holding public hearings for ESL populations would also require bilingual assistance, and in most 
programs there are numerous language groups represented. 

3) What problems or other Issues would confront ESL programs when using these or similar Indicators? 

As always, time and resources will be a problem for ESL programs. The activities described above are clearly 
administrative. Adequate resources and technical assistance must be made available if these activities are to be required. 

Part II: Program Process and Content 

1 ) Are there unique aspects of ESL programs that should be taken Into account In the development of Indicators? 
What adjustments and adaptations to a generic set of Indicators may be necessary? 

There is wide diversity among programs offering ESL instruction. This diversity includes type of program, program focus, 
program approaches and philosophies, funding sources, etc. There is also a wide diversity in learner goals and needs. 
This diversity from both programming and learner needs perspective must be taken into account. There must be a 
balance between the need to control and regulate quality services and the need to allow for creative, innovative and 
responsive programming. 

2) Are these Indicators appropriate for ESL programs? Are any unnecessary?, Should any be added? Which ones 
and why? 

ADJUST: 

The indicator that will be the most problematic for ESL programs and learners at the lowest proficiency levels is 
"development of Individual learning plans". It is not the process that will be challenged, but the product. Many 
practitioners will tell you that at this level "an individual learning plan" is an empty exercise in paperwork. Most 
practitioners will agree that learners at the lowest proficiency levels cannot express goals in English and indeed the 
concept is likely to be culturally alien to them. Forcing very beginning language learners (particularly literacy students) 
to set goals that may be too high and unrealistic can frustrate them and cause them to drop out. However, most 
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practitioners recognize that it is important to help learners take charge of their learning by enabling them to begin to 
discover and articulate their goals. "A process for accomplishing this must keep two issues in mind, (1) learners new to 
speaking English may not bo able to articulate their goals (especially at the beginning course), and (2) learner goals may 
change as they gain greater confidence and increase their proficiency." w For learners at intermediate and advanced 
levels, individual learning plans may be more appropriate instruments for goal definition than for the lower proficient 
students. For them, perhaps more manageable learning goals (ie., life skills competencies) that the learner can achieve 
and experience success with will be more favorably received by both learners and instructors and will enable them to 
begin the process of goaJ articulation. 

3) What problems or other lasuee would confront ESL programs when using these or similar Indicators? 

Discussed in #1 above. 

Part ill: Curriculum and Materials 

1 ) Are there unique aspects of ESL programs that should be taken into account In the development of Indicators? 
What adjustments and adaptations to a generic set of Indicators may be necessary? 

ESL populations, unlike ABE populations are ever changing and programs must ta able to recognize new population 
trends, assess their neeoc and respond with new and appropriate curricula that are linked to the changing needs. It is 
reasonable for a state agency to require programs to define their instructional program in such a way as to enable the 
state to determine whether content is based on learner goals and needs assessment as well as whether the processes 
are appropriate. For example, if a program determines that a large number of prospective learners need to learn English 
in order to function in daily life, a grammar-based syllabus would not show a relationship between learner needs and the 
proposed plan of instruction. Similarly, a program that has as its goal to help parents support the schooling of children, 
needs a curriculum that includes strategies for understanding and interacting with the U.S. school system. 
Responsiveness to context is the key for determining appropriateness of curriculum and approaches. 



2) Are these indicators appropriate for ESL programs? Are any unnecessary?, Should any be added? Which ones 
and why? 

The indicators are appropriate. 

Most adult ESL programs still do not have access to adequate equipment for instruction. As our communities and work 
places become more technologically advanced, it is increasingly important for adult learners to use educational 
technologies. Adult ESL programs should begin to at least plan for the use of educational technologies and explore ways 
in which the equipment might be acquired. 

SUGGESTED ADDITION: 

a) Methods and materials should be educationally sound and reflect an understanding of how adults learn 

b) Program has developed a plan for using and integrating educational technologies 

3) What problems or other Issues would confront ESL programs when using these or similar Indicators? 

Time and resources will be required to truly achieve this standard. ESL teachers are very creative individuals and the field 
would benefit from additional allocation of resources to programs for classroom based research, teacher H <put in 
programming and evaluation, and development and dissemination of materials. 

Part IV: Staff Qualifications 

. , Ve there unique aspects of ESL programs that should be taken Into account In the development of Indicators? 
What adjustments and adaptations to a generic set of Indicators may be necessary? 

All of the indicators specified by Pelavin are commendable and would indeed demonstrate quality in the area of staff 
qualifications. However, there is a unique aspect that should be taken into account - the largely part time staffing situation 
of most ESL programs. This severely limits their availability for staff development and for more staff input and involvement 
in program design and evaluation. Additional funds for these activities must be forthcoming. 

2) Are these indicators appropriate for ESL programs? Are sny unnecessary?. Should any be added? Which ones 
and why? 
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The indicators are appropriate with adequate resources to support them. Paid pre-sorvice and inservice as weii as pay 
for non-classroom program input are essential elements of quality indicators for staff development. 

Suggested Addition: 

a) The term "Staff qualifications" as opposed to "staff characteristics" should be used in the indicators. There is a 
specialized body of knowledge about teaching English as a Second Language that has been developed and articulated. 
ESL teachers should have training in the study of language, second language acquisition theory, adult learning theory 
and ESL teaching methodology. 

b) Paid Planning time 3houid be included as an indicator of program quality. Paying for planning time recognizes this as 
a professional activity and is consistent with practices in other educational settings such as the public schools. 

c) Use of volunteer staff should be viewed as a supplement to paid instructional staff. 

3) What problems or other Issues would confront ESL programs when using these or similar Indicators? 

Most adult ESL programs have chosen to allocate funds toward sen/ices at the expense of full time staffing. This is due 
to the burgeoning need for services that administrators face daily. Most ESL programs could not function without the hard 
work and contributions of a trained part-time workforce and a volunteer force. However, if programs are to be measured 
by new and stringent quality indicators, it must be made clear to the funding agency that professional standards require 
a stable and professional base staff. As Hal Beder states, "Clearly, if adult literacy education is worth doing, it is worth 
doing well. This requires a well trained, weii paid professional workforce. To this end, reliance on part time teachers and 
volunteers is anathema." 21 

Section III: Program Outcomes 

1 ) Are there unique aspects of ESL programs that should be taken Into account In the development of indicators? 
What adjustments and adaptations to a generic set of Indicators may be necessary? 

Limited time and resources, open entry/open exit procedures (which are typical of most ABE/ESL programs), and lack 
of good standardized test measures, diversity of program types, settings, and goals are all aspects that should be taken 
into account when setting indicators. Wrigiey and Guth suggest that: "Given the diversity of programs and the resources 
that have been expended on ESL, it may be unreasonable to expect programs to meet externally defined standards. It 
may first be necessary to require programs to make all reasonable efforts to (1) improve Vieir services (many may need 
guidance) and (2) provide documentation and what they have done and why, and then (3) develop their own standards 
given their particular realities." 2 ' 

2) Ale these Indicators appropriate for ESL programs? Are any unnecessary?, Should any be added? 
Which ones and why? 

Indicators in the area of retention, educational gains and learner goals are appropriate, indicators in the area of 
employment changes are not appropriate for general aduit ESL programs, because of reasons described in rationale 
section. An additional problem for ESL programs in inquiring about employment is the issue of confidentiality. Many 
programs are funded by non federal dollars and there are probably many undocumented aliens participating in the 
programs who would be reluctant to give employment data. It would be disruptive to program procedures to require 
employment information for some of the students and not from others, particularly when there is very little confidence in 
the information's accuracy and relevance. 

3) What problems or other Issues would confront ESL programs when using these or similar Indicators? 

Standardized testing remains a problem for adult ESL programs. Although there is a federal requirement for standardized 
tasting, it should be recognized that there are no standardized tests which can be generally applicable to all types of ESL 
programs. Most are not sensitive enough to measure general language proficiency gains in short periods of time 
(especially for low proficiency levels), and most are very lengthy and costly to administer. Additionally, most standardized 
tests focus on individual facets of language, listening, speaking, reading, writing « x ; grammar. This one dimensional 
approach does not capture the holistic naturo of second language learning. Thor.ias Stlcht, in his report, "Testing and 
Assessment in ABE and ESL Programs," reports that there is serious concern from the field about the federal requirement 
for standardized testing. 23 He suggests, "Generally, in testing in ESL programs, as in other ABE programs, it may be 
desirable to separate testing for program accountability from testing for instructional decision making, -24 This is a 
valuable suggestion and programs should be allowed to submit other data on learner gains which will yield more 
meaningful data related to the outcomes of instruction. 

Learner gains in ESL should also be considered in light of various program and student related conditions. Research 
conducted by the MELT project identified program related factors such as intensity of instruction offered, program 
curricula, trained staff, etc had an impact on amount of learner gains made. While these quality indicators will help 
address some of these program factors, there are other student related factors which impact on learning gains (age, 
previous education, previous language experience, physical abilities, etc.) 2 * 
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